2,088 research outputs found
Extending Hybrid CSP with Probability and Stochasticity
Probabilistic and stochastic behavior are omnipresent in computer controlled
systems, in particular, so-called safety-critical hybrid systems, because of
fundamental properties of nature, uncertain environments, or simplifications to
overcome complexity. Tightly intertwining discrete, continuous and stochastic
dynamics complicates modelling, analysis and verification of stochastic hybrid
systems (SHSs). In the literature, this issue has been extensively
investigated, but unfortunately it still remains challenging as no promising
general solutions are available yet. In this paper, we give our effort by
proposing a general compositional approach for modelling and verification of
SHSs. First, we extend Hybrid CSP (HCSP), a very expressive and process
algebra-like formal modeling language for hybrid systems, by introducing
probability and stochasticity to model SHSs, which is called stochastic HCSP
(SHCSP). To this end, ordinary differential equations (ODEs) are generalized by
stochastic differential equations (SDEs) and non-deterministic choice is
replaced by probabilistic choice. Then, we extend Hybrid Hoare Logic (HHL) to
specify and reason about SHCSP processes. We demonstrate our approach by an
example from real-world.Comment: The conference version of this paper is accepted by SETTA 201
Accelerated Model Checking of Parametric Markov Chains
Parametric Markov chains occur quite naturally in various applications: they
can be used for a conservative analysis of probabilistic systems (no matter how
the parameter is chosen, the system works to specification); they can be used
to find optimal settings for a parameter; they can be used to visualise the
influence of system parameters; and they can be used to make it easy to adjust
the analysis for the case that parameters change. Unfortunately, these
advancements come at a cost: parametric model checking is---or rather
was---often slow. To make the analysis of parametric Markov models scale, we
need three ingredients: clever algorithms, the right data structure, and good
engineering. Clever algorithms are often the main (or sole) selling point; and
we face the trouble that this paper focuses on -- the latter ingredients to
efficient model checking. Consequently, our easiest claim to fame is in the
speed-up we have often realised when comparing to the state of the art
Model checking Ï-regular properties for quantum Markov chains
© Yuan Feng, Ernst Moritz Hahn, Andrea Turrini, and Shenggang Ying. Quantum Markov chains are an extension of classical Markov chains which are labelled with super-operators rather than probabilities. They allow to faithfully represent quantum programs and quantum protocols. In this paper, we investigate model checking !-regular properties, a very general class of properties (including, e.g., LTL properties) of interest, against this model. For classical Markov chains, such properties are usually checked by building the product of the model with a language automaton. Subsequent analysis is then performed on this product. When doing so, one takes into account its graph structure, and for instance performs different analyses per bottom strongly connected component (BSCC). Unfortunately, for quantum Markov chains such an approach does not work directly, because super-operators behave differently from probabilities. To overcome this problem, we transform the product quantum Markov chain into a single super-operator, which induces a decomposition of the state space (the tensor product of classical state space and the quantum one) into a family of BSCC subspaces. Interestingly, we show that this BSCC decomposition provides a solution to the issue of model checking Ï-regular properties for quantum Markov chains
PrIC3: Property Directed Reachability for MDPs
IC3 has been a leap forward in symbolic model checking. This paper proposes
PrIC3 (pronounced pricy-three), a conservative extension of IC3 to symbolic
model checking of MDPs. Our main focus is to develop the theory underlying
PrIC3. Alongside, we present a first implementation of PrIC3 including the key
ingredients from IC3 such as generalization, repushing, and propagation
Parameter-Independent Strategies for pMDPs via POMDPs
Markov Decision Processes (MDPs) are a popular class of models suitable for
solving control decision problems in probabilistic reactive systems. We
consider parametric MDPs (pMDPs) that include parameters in some of the
transition probabilities to account for stochastic uncertainties of the
environment such as noise or input disturbances.
We study pMDPs with reachability objectives where the parameter values are
unknown and impossible to measure directly during execution, but there is a
probability distribution known over the parameter values. We study for the
first time computing parameter-independent strategies that are expectation
optimal, i.e., optimize the expected reachability probability under the
probability distribution over the parameters. We present an encoding of our
problem to partially observable MDPs (POMDPs), i.e., a reduction of our problem
to computing optimal strategies in POMDPs.
We evaluate our method experimentally on several benchmarks: a motivating
(repeated) learner model; a series of benchmarks of varying configurations of a
robot moving on a grid; and a consensus protocol.Comment: Extended version of a QEST 2018 pape
Model-free reinforcement learning for stochastic parity games
This paper investigates the use of model-free reinforcement learning to compute the optimal value in two-player stochastic games with parity objectives. In this setting, two decision makers, player Min and player Max, compete on a finite game arena - a stochastic game graph with unknown but fixed probability distributions - to minimize and maximize, respectively, the probability of satisfying a parity objective. We give a reduction from stochastic parity games to a family of stochastic reachability games with a parameter Δ, such that the value of a stochastic parity game equals the limit of the values of the corresponding simple stochastic games as the parameter Δ tends to 0. Since this reduction does not require the knowledge of the probabilistic transition structure of the underlying game arena, model-free reinforcement learning algorithms, such as minimax Q-learning, can be used to approximate the value and mutual best-response strategies for both players in the underlying stochastic parity game. We also present a streamlined reduction from 112-player parity games to reachability games that avoids recourse to nondeterminism. Finally, we report on the experimental evaluations of both reductions
Reachability in Parametric Interval Markov Chains using Constraints
Parametric Interval Markov Chains (pIMCs) are a specification formalism that
extend Markov Chains (MCs) and Interval Markov Chains (IMCs) by taking into
account imprecision in the transition probability values: transitions in pIMCs
are labeled with parametric intervals of probabilities. In this work, we study
the difference between pIMCs and other Markov Chain abstractions models and
investigate the two usual semantics for IMCs: once-and-for-all and
at-every-step. In particular, we prove that both semantics agree on the
maximal/minimal reachability probabilities of a given IMC. We then investigate
solutions to several parameter synthesis problems in the context of pIMCs --
consistency, qualitative reachability and quantitative reachability -- that
rely on constraint encodings. Finally, we propose a prototype implementation of
our constraint encodings with promising results
Incremental Verification of Parametric and Reconfigurable Markov Chains
The analysis of parametrised systems is a growing field in verification, but
the analysis of parametrised probabilistic systems is still in its infancy.
This is partly because it is much harder: while there are beautiful cut-off
results for non-stochastic systems that allow to focus only on small instances,
there is little hope that such approaches extend to the quantitative analysis
of probabilistic systems, as the probabilities depend on the size of a system.
The unicorn would be an automatic transformation of a parametrised system into
a formula, which allows to plot, say, the likelihood to reach a goal or the
expected costs to do so, against the parameters of a system. While such
analysis exists for narrow classes of systems, such as waiting queues, we aim
both lower---stepwise exploring the parameter space---and higher---considering
general systems.
The novelty is to heavily exploit the similarity between instances of
parametrised systems. When the parameter grows, the system for the smaller
parameter is, broadly speaking, present in the larger system. We use this
observation to guide the elegant state-elimination method for parametric Markov
chains in such a way, that the model transformations will start with those
parts of the system that are stable under increasing the parameter. We argue
that this can lead to a very cheap iterative way to analyse parametric systems,
show how this approach extends to reconfigurable systems, and demonstrate on
two benchmarks that this approach scales
Probabilistic Guarantees for Safe Deep Reinforcement Learning
Deep reinforcement learning has been successfully applied to many control
tasks, but the application of such agents in safety-critical scenarios has been
limited due to safety concerns. Rigorous testing of these controllers is
challenging, particularly when they operate in probabilistic environments due
to, for example, hardware faults or noisy sensors. We propose MOSAIC, an
algorithm for measuring the safety of deep reinforcement learning agents in
stochastic settings. Our approach is based on the iterative construction of a
formal abstraction of a controller's execution in an environment, and leverages
probabilistic model checking of Markov decision processes to produce
probabilistic guarantees on safe behaviour over a finite time horizon. It
produces bounds on the probability of safe operation of the controller for
different initial configurations and identifies regions where correct behaviour
can be guaranteed. We implement and evaluate our approach on agents trained for
several benchmark control problems
- âŠ